Method for inlaying images or video within another video sequence

ABSTRACT

A method for inlaying images within another video sequence comprises: controlling the acquisition of the main sequence and recording it in a memory of mobile equipment; simultaneously controlling, at the initiative of the user, a) the display of the main sequence in a first part of the display screen of the mobile equipment; b) acquiring the secondary video sequence with at least one image sensor of the mobile equipment; c) recording the secondary video sequence in the memory of the mobile equipment; d) displaying the secondary video sequence in a second part of the display screen of the mobile equipment, in a synchronous manner with the display of the main sequence; transmitting the recordings over a server via a communication session; automatically controlling a step of inlaying the secondary video sequence ink the main sequence; controlling the recording of the merged sequence thus produced on the server; and computing an access link by an item of equipment remote from the merged sequence.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national phase entry under 35 U.S.C. § 371 ofInternational Patent Application PCT/FR2018/051167, filed May 14, 2018,designating the United States of America and published as InternationalPatent Publication WO 2018/211207 A1 on Nov. 22, 2018, which claims thebenefit under Article 8 of the Patent Cooperation Treaty to FrenchPatent Application Serial No. 17/54395, filed May 18, 2017.

TECHNICAL FIELD

The present disclosure relates to the general field of image processing,in particular, video sequences on mobile terminals. It more particularlyrelates to a method for inlaying images or video within another sequenceof images or video.

BACKGROUND

As regards mobile video, the use of a cellular phone (“smartphone”) or atablet becomes widespread. One of the most widely used applications ofconnected mobile equipment is the creation of self-portraits of the“selfie” type or videos instantly broadcast on social networks or madeavailable to third parties via a public or private server.

Some have also devised applications to be present on both sides of thelens at the same time, for example, to capture a landscape together withfaces to share them with friends. Such an application is available onthe digital store “app Store” under the trade name “Selfie Vista”.

When the “Selfie Vista” application is launched, the front camera isactive and displayed on the screen, but when a button associated withthe dual recording function is pressed, the image is captured with thephoto cell both front and back simultaneously.

Solutions for recording images from two sensors of the same connectedmobile equipment are known from the state of the art.

The U.S. Pat. No. 8,866,970 describes, for example, a method forinlaying an image to be inlaid in a video sequence, for a mobileterminal such as a tablet or a smartphone, which includes the followingsteps:

-   -   selection of an image to be inlaid,    -   playback of the video sequence,    -   display of the frame,    -   determination of the presence of an inlay zone on the frame, the        inlay zone being previously identified, or deduced according to        a predefined algorithm,    -   application of a deformation on the image to be inlaid, so as to        make this image to be inlaid coincide with the shape of the        inlay zone,    -   display of the image combining the frame and the image to be        inlaid, arranged in the inlay zone.

This solution offers a video sequence processing method on mobileterminals, more precisely, a real-time image inlay in the video stream.After computing the inlay points, the video stream is played back withthe inlaid images. In order to perform the inlay, the image is merged tothe video stream, frame by frame. For each frame, the image ispositioned in the right place. This method involves the image undergoinga trapezoidal matrix transformation in real time so that it can adapt tothe video stream.

The patent application EP2289242 describing a method for providing videocontent is also known, which includes the following steps:

-   -   establishing a video session between a mobile terminal and a        remote communication system;    -   provision to the terminal, by the communication system, of a        video enriched with one or more markers inlaid in the video;        activation from the terminal of at least one marker during the        video session    -   sending by the communication system to the mobile terminal of a        text message (e. g. SMS or e-mail) containing a bookmark        associated with the activated marker.

Prior art solutions are not fully adapted to implementation on mobileequipment with limited resources for both storage and real-timecalculation processing.

These solutions generally require real-time processing of videosequences at high resolution, which requires both very high RAM capacityto allow image merging calculations, as well as high permanent memorycapacities and powerful graphics processors.

These solutions are therefore only suitable for sophisticated “high-end”terminals.

BRIEF SUMMARY

In order to remedy these disadvantages, the present disclosure in itsbroadest sense relates to a method for the production of video sequencesusing connected mobile equipment, the video sequence comprising a mainsequence or an image and an inlay of a secondary sequence, characterizedin that it comprises the following steps:

-   -   controlling the acquisition of the main sequence and its        recording in a memory of the mobile equipment, controlling, at        the initiative of the user,        -   a) the display of the main sequence in a first part of the            display screen of the mobile equipment        -   b) acquiring the secondary video sequence with at least one            image sensor of the mobile equipment,        -   c) recording the secondary video sequence in the memory of            the mobile equipment        -   d) displaying the secondary video sequence in a second part            of the display screen of the mobile equipment, in a            synchronous manner with the display of the main sequence,    -   transmitting the recordings over a server via a communication        session,    -   automatically controlling a step of inlaying the secondary video        sequence in the main sequence to create a merged sequence,    -   controlling the recording of the merged sequence thus produced        on the server, and computing an access link by an item of        equipment remote from the merged sequence.

According to a first alternative solution, the method also includes astep of transmitting the merged sequence to the connected mobileequipment during the communication session.

According to a second alternative solution, the method also includes astep of transmitting the link of the merged sequence to a server hostinga third-party streaming application.

According to a third alternative solution, the method according to thepresent disclosure also includes a step of transmitting the link of themerged sequence to a server hosting a third-party video downloadapplication.

BRIEF DESCRIPTION OF THE DRAWINGS

The characteristics and advantages of embodiments of the presentdisclosure will be better appreciated through the following description,which sets out the characteristics of the present disclosure through anopen-ended example of application.

The description is based on the appended figures representing:

FIG. 1 is a schematic view of the screen of a connected item ofequipment during the acquisition phase of the second sequence.

FIG. 2 is a schematic view of the screen of a connected item ofequipment during the playback phase of the sequence merged with theinlay.

FIG. 3 is an organization chart of the steps involved in the presentmethod.

FIG. 4 is an organization chart of the steps for creating and revising aSelfyoo file.

DETAILED DESCRIPTION

The implementation of the present disclosure is based on connectedmobile equipment, for example, a “smartphone” or a tablet, and a serverto which the mobile equipment is connected.

To enable the mobile equipment to run the application to merge twoseries of images or videos, an application is first installed on themobile equipment, for example by downloading a file containing theexecutable code from a download platform.

This code is compatible with the operating system of the mobileequipment, for example Android or iOS, and makes it possible for thesoftware and hardware of the host mobile equipment to be controlled forenforcing the specific functionalities of the present disclosure.

To produce a sequence according to the present disclosure, an image or avideo present in one's smartphone is first recorded or used, i.e.,either a still image or a video sequence, which will constitute the mainsequence. This sequence may be a view of a landscape during a touristtrip, or a sequence taken inside an apartment or a house that is desiredto be rented or sold.

The next step consists in activating the application to control thedisplay of a thumbnail 1 corresponding to an image of the main sequencein a part of the screen.

This thumbnail 1 is graphically associated with a time scroll bar 2 andan elapsed time counter 3.

A touch button 4 enables the user to control the start of the recordingof a second video sequence from the front sensor 5 of the connectedmobile equipment, which is displayed as a display zone 6 appearing inanother part of the screen.

When the recording of the second sequence is satisfactory, the useractivates the transmission to the server by means of a touch button 7controlling the opening of a session and the transfer of the filescorresponding to the first and second recordings.

The server then inlays the second video file into the image or the firstvideo file, for example with a software application whose trade name isFF MPEG Merge.

The resulting file is recorded on the server and transmitted to theconnected mobile equipment to allow a display, an example of which isshown in FIG. 2. On the screen, the content is presented as a singlemoving image formed by the main image 8 and a part 9 in which therecording of the second sequence appears.

Organizational Chart

The first step 10 consists in recording or using a still image or avideo sequence with the back sensor of the connected mobile equipmentand recording the file in a digital format, for example Mpeg, in the RAMof the mobile equipment.

The second step 11 consists in recording a second video sequence with atleast one image sensor of the connected mobile equipment and recordingthe file in a digital format, for example Mpeg, in the RAM of the mobileequipment.

These two files are then transmitted to the server by transfer steps 12,13.

The server then performs image processing 14 for inlaying the videosusing an application such as FF MPEG Merge (trade name).

The resulting digital file is transmitted by a step 15 to the connecteditem of equipment and/or by a step 16 to a download or streaming server17.

Selfyoo is an application that makes it possible to create advancedvideo selfies based on an overlay of videos.

In local situations (i.e., when the processing is performed on theequipment itself), it is impossible to create an overlay based on theoriginal Android video player.

The company has developed an innovative and unique technologicalsolution, thanks to a modification of the source code of the Androidand/or IOS reader.

In addition to the basic video overlay, a non-cubic shape couldsuccessfully be added to the overlaid video selfie, thanks to a“transparent filter”.

In the server/user situation (i.e., when the processing is performed onthe server and sent back to the equipment), a Java 1.8.0 application hasbeen developed with the use of the FFmeg library (video editing tool)that gives the same possibilities as in a local situation (i.e.,overlaid videos and a non-cubic frame).

The advantage of the server-based processing (Linux Ubuntu 14.0) is theaddition of a motion capability of the overlaid video selfie.

From a technological point of view, the films have been divided intoisolated images, the location of the elements for each image has beenstudied, and these have been reassembled into a single film.

An additional advantage is the storage of data in a MySQL 5.7.16database.

FIG. 4 shows the system and the steps for creating and reviewing aSelfyoo file, which is an overlay of a video in a media database. Thesystem includes, depending on the operation, a transmitter, a server anda recipient. The transmitter is a connected mobile item of equipment 41,for example, a “smartphone”, which performs the steps E1-E6. The step E1aims to select a media database. The media database can be a photo or avideo, which is used as a wallpaper, over which another video can beoverlaid. This selection can be made from photos or videos that arealready saved in the phone photo/video gallery. It is also possible totake a picture or video directly with the Selyfoo application. When avideo is selected, the video is compressed, either in real time duringvideo shooting or after selection from the phone video gallery. Afterthe user has completed step El and selected the media database, thetransmitter starts step E2 to send it from the transmitter to theserver, which is a work done in the background. On the transmittingequipment, the user then performs step E3 for recording an overlayvideo, which will be inlaid as an overlay in the media database.

Further to step E3, the Selfyoo application performs step E41 byrequesting the creation of a Selfyoo identifier on the server. Therequest returns the Selfyoo identifier from the server to thetransmitter. At the same time, the media database and the inlay videoare locally saved on a phone folder and available on the “Selfyoogallery”.

The step E4 of sending the inlay video to the server is performed inparallel with the step E41 by the Selfyoo application on the phone inthe background.

After the creation of the Selfyoo identifier in step E41, step E42performs a deep link creation and a possibility to share this link. Thislink gives access to the Selfyoo file, in which the overlay video isinlaid in the media database.

Step E5 is about sharing the deep link through a third-partycommunication application or a social network, for example, Whatsapp,Facebook, email, etc. It is also possible to share the deep link throughthe Selfyoo application, which is itself a social sharing network. StepE6 gives the possibility to locally review the Selfyoo file on thetransmitter.

Between the server 42 and the transmitter 41, the dotted lines representthe communications over the network. In the step S1, the server receivesthe media database from the transmitter. In step S2, the server returnsthe Selfyoo identifier created as a result of the request in step E41.In step S3, the server saves the overlay video sent in step E4. In stepS4, the overlay video is inlaid in the media database to create aSelfyoo file, for which the wallpaper is the media database, as shown inFIG. 2.

A review is done on a recipient, which is a connected mobile item ofequipment either with the Selfyoo application 43 or without the Selfyooapplication 44. Between the recipients and the transmitter, the deeplink is transmitted by a third party application such as Whatsapp,email, etc., as indicated by the dotted lines.

On the recipient with the Selfyoo application, in step DA1, the mobileitem of equipment receives the link through its third-party applicationor via the Selfyoo application. In step DA2, the user clicks on thelink, which opens the Selfyoo application. In step DA3, the Selfyooapplication sends a playback request to the server, which triggers thedownload of the media database and the overlay video. In step DA4, theSelfyoo application on the recipient recreates the Selfyoo file byre-inlaying the overlay video in the media database in real time. Thisgives the possibility to move the overlay video, which is in a smallinlay window in the media base that takes up the entire screen surface.If the media database is a photo, this function gives the possibility toposition the video in different places in the photo to synchronize thecomments in the video with specific places in the photo. If the mediadatabase is a video, this function makes it possible to synchronize thecontents of two videos.

On the Selfyoo application of the recipient, it is also possible toreverse the roles of the media database and the overlay video, duringthe recreation of the Selfyoo file to the recipient. In this case, theprior overlay video takes up the entire screen surface, whereas theprior media database is positioned in a small Selfyoo inlay window.

On a recipient without the Selfyoo application, in step DS1, the userreceives the deep link through a third-party application, for example,Whatsapp, email. In step DS2, the user clicks on the link and receivesthe Selfyoo file created in step S4, in which the inlay video is alreadyinlaid in the media database. The Selfyoo file with an inlay video,which is an overlay video frozen on a wallpaper, is presented on a videoplayer opened by default on the recipient, for example: browser, VLC,video player. This function does not make it possible to freely move theoverlay video in the media database, nor to reverse the roles of theoverlay video and the media database.

1. A method for production of a video sequence using connected mobileequipment, the video sequence comprising a main sequence and an inlay ofa secondary video sequence, comprising the following steps: controllingacquisition of the main sequence and its recording in a memory of themobile equipment, controlling, at an initiative of a user, e) a displayof the main sequence in a first part of the display screen of the mobileequipment, f) acquiring the secondary video sequence with at least oneimage sensor of the mobile equipment, g) recording the secondary videosequence in the memory of the mobile equipment, and h) displaying thesecondary video sequence in a second part of the display screen of themobile equipment, in a synchronous manner with the display of the mainsequence, transmitting the recordings over a server via a communicationsession, automatically controlling a step of inlaying the secondaryvideo sequence in the main sequence to create a merged sequence,controlling the recording of the merged sequence thus produced on theserver, and computing an access link by an item of equipment remote fromthe merged sequence.
 2. The method of claim 1, wherein the acquisitionof the main sequence is carried out by controlling at least one imagesensor of the mobile equipment to capture a first image or a first videosequence of a determined duration.
 3. The method of claim 1, wherein theacquisition of the main sequence is carried out by selecting a photo ora video from a photo or video gallery of the mobile equipment.
 4. Themethod of claim 1 further comprising a step of transmitting the mergedsequence to the connected mobile equipment during the communicationsession.
 5. The method of claim 1, further comprising a step oftransmitting the link of the merged sequence to a server hosting athird-party streaming application.
 6. The method of claim 1, furthercomprising a step of transmitting the link of the merged sequence to aserver hosting a third-party video download application.